Are clinical impressions of adolescent substance use accurate?

OBJECTIVE To compare providers' impressions of adolescents' level of substance use with diagnostic classifications from a structured diagnostic interview. METHODS Secondary analysis of data was conducted from a validation study of the CRAFFT substance abuse screening test of 14- to 18-year-old medical clinic patients (n = 533) and their corresponding medical care providers (n = 109) at an adolescent clinic affiliated with a large tertiary care pediatric hospital. Medical care providers completed a form that recorded their clinical impressions of patients' level of alcohol and drug involvement (none, minimal, problem, abuse, dependence) and demographic characteristics. The form included brief diagnostic descriptions for each level of use. After the medical visit, patients completed the Adolescent Diagnostic Interview (ADI), a structured diagnostic interview that yields diagnoses of abuse and dependence according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV). On the basis of their past 12 months of alcohol and drug use on the ADI interview, adolescents were classified into 5 mutually exclusive diagnostic groups. "None" was defined by no reported use of alcohol or drugs during the past year. "Minimal use" was defined as use of alcohol or drugs but no report of any substance-related problems. "Problem use" was defined as reporting 1 or more substance-related problems but no diagnosis of abuse or dependence. "Abuse" was defined by meeting any 1 of 4 DSM-IV diagnostic criteria for either alcohol or drug abuse but no diagnosis of dependence. "Dependence" was defined by meeting any 3 of 7 diagnostic criteria for either alcohol or drug dependence, with or without a diagnosis of abuse. Proportions were compared using Fisher exact test. Agreement was assessed with the weighted kappa, and these analyses were stratified by substance used (ie, alcohol vs drug) and demographic characteristics. Sensitivity, specificity, and positive and negative predictive values were calculated from 2 x 2 tables. RESULTS Compared with the criterion standard interview, providers identified significantly fewer patients with problem use and abuse and no patients with dependence. Of >100 patients whom the ADI classified with substance problem use, providers correctly identified 18. Of 50 patients who were classified with a diagnosis of alcohol or drug abuse, providers correctly identified 10. Of 36 patients who were classified with a diagnosis of alcohol or drug dependence, providers correctly identified none. For the 86 adolescents who were classified with a substance-related disorder (ie, abuse or dependence), providers' impressions were "none" (24.4%), "minimal use" (50%), "problem use" (15.1%), "abuse" (10.5%), and "dependence" (0%). There was only marginal agreement between providers' impressions and diagnoses related to alcohol use (kappa = .29), drug use (kappa = .31), and any substance use (kappa = .30). Kappa was not significantly affected by the patient's age, but it was by gender. Among boys, kappa was significantly higher for impressions of drug use versus alcohol use (kappa = .48 vs kappa = .27); and, among drug users, kappa was significantly higher among boys compared with girls (kappa = .48 vs kappa = .24). Kappa did not differ significantly across race/ethnicity subgroups, although there is a suggestive trend toward higher agreement for black non-Hispanic compared with white non-Hispanic adolescents (kappa = .35 vs kappa = .21). Kappa did not differ significantly on the basis of the visit type, but the size of this difference (kappa = .36 vs kappa = .24) suggests that the longer well-child visit yielded greater identification of substance-related pathology. Providers' impressions had a sensitivity of .63 for identifying use of alcohol or drugs. However, sensitivity was poor for identification of problem use (.14), abuse (.10), and dependence (0), whereas specificity and positive predictive values were high. Of the 86 adolescents with a diagnosis of abuse or dependence, 75.6% were correctly identified by providers as using substances; however, the level of use in 50% of these adolescents was reported by providers as minimal. CONCLUSIONS In this study, clinical impressions of adolescents' alcohol/drug involvement underestimated substance-related pathology. When providers thought that use was present, there was a very high likelihood that a problem or disorder existed. The use of structured screening devices would likely improve identification of adolescents with substance-related pathology in primary care settings and should be considered for use with all adolescent patients, rather than only those who are perceived to be at higher risk.